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=== (54) Title: OPERATION METHOD OF USER INTERFACE OF HAND-HELD DEVICE 




(57) Abstract: An operation method of a user interface of a handheld device, like a palm-size or pocket-size PC, mobile telephone, 
communicator or equivalent, in which method data on a display of the device is changed for finding or selecting desired data, is 
characterized in that data (5a - 5e) on the display (2*) of the device is changed responsive to certain movements (F) of the device in a 
three-dimensional space of use thereof, and said movements of the device and the corresponding changes of the data on the display 
are defined so that a movement and a corresponding change of data on the display have a cause-consequence relationship which is 
analogous to cause-consequence relationships of the three-dimensional space of perception and action of a user (1). 
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Operation method ofussr interface of hand-held device 

The invention is related to an operation method of the user interface of a hand-held 
data-processing device. A hand-held device means here apainb-top or pocket comp ute r,. 
5 mobile phone, camrnnmcator on: like device, essential features of which include displaying 
data on the display of the device and in which data on the display of the device is changed 
far finding and selecting a desired data 

User interfaces of above mentiooed devices are presently mostly of type "point and 
clicks A general operation method of a user interface is browsing and selecting data by 
10 means of keys. Palm-top computers are usually managed with a touch sensitive display, a 
virtual kcyboarc^ a pec, and oftea some auxiliary hard keys or buttons. Firstly, this kind of 
user interfaces are difficult to tt^ibrsmsD size of the devk>es ai^ 
or vicinal keys. On the other hand, the use becnna nro 

caonthiuonsly provided with new applications and services which are to be used with the 
15 same small quantity of small-sized keys. Charactexistric of these user interfaces, like user 
interfaces of computers and other data-processing devfees, is generally also that they have 
their own artificial logics and rules which areTidly adopted and known only by a fetv > 
normally technically orientated users. 

An object of the invention is to present snch tmi operation method of the user interface 
20 of hand-held devices which to larg^ extent removes many of the above mentioned problems. 
To reach this object, the operation irethodof the user interface of ^ 
like palm-top or pocket computer, mobile phone, conrammcator or Eke, according to the 
invention, in which operation method data cm the display of the device is changed for 
finding and selecting a desired data, is characterized in that what is defined by claim 1. 
25 Other claims define variable anhodinKaite ^ invealioflL 

An advantage of the invention is thai a user is able to easily adopt the operating rules of 
the user interface because they are "natural* 1 - The use of the device is also ergCHiomlcally 
easier because there is no need to use small-sized keys car auxiliary devices, for example. 
The invention and some embodiments thereof are described in further detail in the 
30 following with reference to fee accompanying drawings, in which: 

Figs* 1 to 10 Hie schematic perspective views presenting examples of the operation 
method of the user interface of the invention; 

Fig?- 11 to 13 me schematic front views of a device and daia presented on the display 
thereof and present some further examples of the operation method of die uses* interface of 
35 the invention; 



Fig. 14 is a . schemati c presentation of an arrangement of data and operation of an 
application of the user interface of the invention; 

Eg. 15 is a schematic perspective view of operation of a further application of the user 
interface of the invention; and 
5 Fig. 1 6 Is a flo w chart presenting a possible realization of the operation method of the 

nser interface according to the invention. 

In Figs- 1 to 5, a user is holding a hand-held device 2 in his or her hand 3 and is using a 
device having a display T . The display V is a touch sensitive display, and the user is able to 
give a signal to the device by pressing the display with bis or her thumb 4. Movements of 
20 the device 2 are detected by an ecoelcratinn measurement circuit installed in the device 
itself for example, and the data obtained from the circuit is utilized in the operation of the 
user interface . Same possible ways of realizing the operation method of the user interface 
according to the invention are considered in further detail below. Figs. 1 to 5 present 
examples of the operation of the user interface. 
IS With reference to Fig. 1, pages 5a, 5h r 5c, 5d and 5e > wfuchnmy bee. g, weather rrraps 

obtained as a result of a web search, are browsed on the display 2' in response to moving the 
devic* 2 essentially j»en^^ 
u^ nser» a is easy to diirJc ftat lie or she 

and that by moving m the pile he or she is able to ^ a picture at which he or sJw is at each 
2D tnne.bi the same way, sijcces^ 

thought to be btowsoL Tbe speed of hro 

intensity of the movement, he. on the magnitude of acceleration in the movement. The 
operation may also be arranged in such a v^ftiat a suitable smaDoxrveraert 
backward and immediate stop bring the next page m the c^^ 
v display. Furtfaeimca^ fl in^^ 

signal to the display control means in response to which signal changing of the data on the 

display is stopped. This kind of virtual use of depmiiMsreara 

display having a restricted size. 

Wiui jefcew» to F%. 2, an mnag^ on th^ 
smaller, image 6a, in response to nwvuag the device 2 in the sanie^^ 
perpendicularly to the display 2' thereof, as is indicated by arrow Z. This cperation method 
of the user interface is fully analogous to human action as he or she wants to esamme details 
of an object, e,g. an image, more closely. Having found a desired magnification, changing of 
the data on the disphry may be stopped again by a posh of the dtspJknr2% for example. The 
fun ctions of Figs. 1 and 2 may be used afternam^y in a user interface according to 
apphtsition, whereby there is no indishncfion of what is bappernngin response to moving 
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the device in this way. The function of Figs. 1 and 2 may also be combined La the operation 
of a user interface so that, ibc example, data Is first browsed and then ixnymed by means of 
the same movement 

or the other of the fimctrans. A selector of zooming may be a push of a certain area on the 
5 display, for example, whereby changing over from the zooming mode to the browsing mode 
may happen automatically by selection of a desired magnification ratio. 

In Fig. 3, reference sign 7a indicates image data which is significantly larger than the 
display. The capacity of the dfcpiay of a hand held device is not large enough for large 
entities of factual or image data. Iniespanse to lateral movement of the device in essentially 
10 the plane of Ac display^, as is indicated by arrows N^E^S and W, data 7b cm the display is 
changed as if a window were moved above a larger image fotmedby tie image data 7a. 
Having found a desired plaoe^ changing of data 7b an the display may again be stopped by a 
push of the display. It is logical to accomplish the fkoetions of Figs. 2 and 3 together 
whereby the image data may be examined mure thoroughly after zooming by selecting the 
1 5 data in the way described by Fig. 3 . Moreover, e. g „ number selection may be realized in. this 
way without keys by moving the device above a large virtual keyboard and selecting the 
desired numbers one by octkl This function makes it possible to read full-length web pages 
with a palm-top computer, foi example. 

With reference to Fig. 4, in response to a quick rotational movement of the device 2 as 
20 if around an axis formed essentially by file edge 9 thereof; Le. in response to swinging die 
device like taming page of a book, as is indicated by auowP, data 8K *c on the display 
T is changed com^spondicgly, e.g~ a page of en electronic hook is turned- By die direction, 
of the swing, the direction of tearing page is selected 1^ 

a small swing with quick stopping tons one page in the direction of the swing, and thtf in 

25 re^Kxnseto ammemle^ 

stopped by a push of the display. 

In Fig. 5, an application is iltustrafed in which any part 9b of image data 9a forming a 
panorama picture covering the whole sphere of perceptual space (or respectively of image 
data forming a c^Imdrical picture) may be examined at a time. In the opeacation method of a 

30 user interface according to the invention^ in response to moving the device 2 as if moving a 
picture cm a corresponding sphcxk^Isui&oe or in response to changing orientation of the 
device, that part 9b of the image data 9a which corresponds to the orientation of the display 
is obtained an the display 2\ At the same time, stopping md zooming of tte image m 
applied in the way described above. It may be contemplated also that a panorama picture is 

35 rotated by giving a push with a n»vemen£of Aw device to ^ 
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panorama picture is correspondingly kept moving until moving is stopped by a posh of tie 
display, for example. 

Figs. 6 to I Q present embodiments of the user interface of the invention in which data 
ob the display is changed by tilting the device In Fig. 6 y an example is presented in which 
5 data, illustrated by data Dl, on the display 2' of the device 2 is scrolled in the way indicated 
by arrow Rl to the direction to which Che device is tilted in the way indicated by arrow Tl, 
The initial positkm of the device is indicated by broken line and the tilled position by solid 
line. In other woids, in response to tilting the device, the data on the display is rolling or 
naming to the directum to which the device is tilted, which is fully analogous to cause- 
10 <x>nsequsioereaLtioasfe^ of the real world- This may be realized in such a way, for 
example, that the ch a nging velocity is the grater the more the device is being; tOted or is 
tilted from the initial position. Rolling or running of the dafamay be stopped byre turning 
the device to the initial position or by means of a key on the device, for example. 

Fig. 7 presents an embodiment suitable for moving and searching objects on a map, fin 
15 example. On the display 2,' a cursor C is placed which 

the device m the way indicated by arrow T2 causes that data, illustrated by (£rta D2, starts 
moving from the tOting direction towards to the cursor C in the way indicated by arrow M2_ 
In other words, the cursor is moving on the data forming a map, for example, to the direction 
to which the device is tOted. 
2B in the example of Fig. 8, dafaonlhe display 2* of the device 2 is Illustrated by dataD3, 

As the user is tilting the device 2 fium the position indict 

himself or herself, like going closer to the dala on the display, the data on the display i* 
enlarged, which is illustrated and arrow Z2. Id the 

example of Fig. 9, dataD4 on the display is ooaespondis^ty reduced, asis indicted by a 
25 graph illustrating the reduction and drawing away and by arrow 73, by tilting the device 
towards the user in the way indicated by arrow T4. 

The embodiments of Figs. 6± 7, 8 and 9 may be combined in the user interface of a 
device in such a wsy, for example, that there is a key on die device for selecting die 
aeration mode (soiling or 
30 scrolling mode* for example, aid having found the object, changing over to the zooming 

mode is made whereby tilting results in aoooiing the data on the display. Scrolling by tilting 
may,, of course, be combined with zooming in the way described with reference to Fig. 2. 

Fig. 10 presents schematically an embodiment of the user interface of the inveartioo for 
selecting data objects on the display. Data objects D5 to D10 may be altema^ 
35 or selection buttons on a page» for example. In the initial position of the dgvf^e 7. indi cated 
by broken line, data object D5 selected which Is indicated by hatching in the figure. Tilting 



the device in the indict 

tilting, as is indicated by airowMS* and is here moved to data object D9- Having selected 
the desired data object, it may be locked by -a key on the device or by a return movement of 
the device* for example. Foe die sake of simplicity, tilting the device to only one direction, 
and selection from data objects located one below the other arc presented here, but in the 
same way it is, of course, possible to move between selectable objects also laterally. 

Fig. 1 1 pteseaits a further example of the embodiment described with reference to Fig, 
3- Wide data D1I consists of objects wbicb are described by letters arranged to form a 
matrix- By moving the device in the way indicated by arrows N, S and Wane is as if 
moving abow the data and looking display^-. As the 

desired daJta, here lell^q, is fminct it maybe selected by means of a key on the device, for 
example. A comparable function may be realized also so that, instead of moving the device 
laterally, it is tilled to a direction to which ane desires to move oa the data. 

Fig. 12 fxsse^ a further 
selecting an object on the display. On the display 2* of the device 2 there are objects OI to 
09 from which the selection is made. In the initial state at left, the cursor is close to the 
upper right comer of the display. In tins embodiment, cursor C is as if anchored in place in 
the real world. As one deshes to select object 07, the device 2 is moved laterally as if 
rnoving a picture under the ctosor so that the desired object OT is coming undo: the cursor C 
at right in the figure. There are keys 21> 22 and 23 on the devke the operation o 
correspond to the operation of foe buttons of a mouse. Inlact, the embodiments of Figs. 10 
to 12 are solutions which in hand-held devices replace solutions, like a mouse, in 
conventional computers which are needed lbr moving, moving a cursor or making selections 
on the display. 

Pig. 13 presents ana eanbodimc^ ... ,CL> 

, Cnare arranged ia a stacfc of the device, like 

e.g, pages of a book in the real workL The data objects may be cards presenting articles of 
commerce or wet> pages^ for example The objects are browsed in the way indicated by _ 
arrow M4 forward and backward by tiffing the device correspondin gly either forward in the 
way indicated by arrow T6 or back ward in the way indicated by anew T7. By means of fays 
24 to 26, objects may be selected and files, programs; etc. behind them be opened. These 
may be several modes in the user interface of a device, like browsing, scrolling, selecting 
and 2xxMning described above* and eg. key 24 may be for selection of mode and other keys 
25, 26 and 27 may operate Eke the buttons of a conventional mouse. 

Fig. 14 presents an application of the user interface of the invention in which data 
objects are arranged in radial stacks SI 1 to S18 to extend outwards from the circle of the 



centre in. which a device 2 ami a user are. The first page of each stack, cg.P12 arP16, 
defines data ttuerebchimL Front pages may be browsed by turning the device laterally in Che 
way indicated by arrow T8^ fbr example, so that direction A12 and front page P12 are 
selected. The stack of data objects behind it may be browsed either by moving or lilting the 
device 2 forward and backward in the way indicated by arrow FI . In this way. e^ a user 
interlace for electronic shopping may be arranged and make perceiving the places of tie 
products and orientation in a virtual shop easier. 

Fig. 1 5 presents a further developed application of the embodiment described with 
reference to Kg. S. Circles Rl and R2 present now image data farming a panorama pictae 
car a 3D picture* in general, covering the whole sphere of the perception space. The image 
data may be also image data obtained from a vide>o camera <xmiiect£d to the device. Three- 
dimensional image data D12 describipga thing, for example; may be brought to this image 
data and set in a desired place and desired position in J *tbe real suuounifiogs". In response to 
moving the device 2 in the way described with reference to Fig. 5, e_g. as indicated by arrow 
T9 > bo€h the background data and 3D dat&D12*,D12" describing the thing are ciranged in 
accordance with the positions of the device to correspond to viewing directions Al and A2, 
so that the thing may be examined from different directions in T tfae ieal surrounding^* 
thereof This may be applied e^_ by searching the 3D model of a 9ofk from a web shop and 
by placing it in a desired place in aa image data presenting one's house whereby it may be 
examined from different directions to see bow the sola looks Kkc in the real surroundings 
thereot 

A solution according to the invention may be realized by providing a hand-held device 
by a multi-axial acceterometer, for example, and with suitable circuits and programs co- 
operating with the operating system of the device and possibly application programs far 
processing and interpreting measuiem^ 

corresponding to a movement detected by theaocelerometer is carried out. Rg_ the 
realization with measurement of accelerations is based on application of as such known 
technical solutions, and a person skilled in the art, having provided with instructions and 
specifications* is able to realize the operation of the user interface according to die invention 
with reasonable efforts. 

In small-sized dafa-procesang devices, like palm-top computers, with prior art 
hardware technology* it is not possible to ie&Hze operation systems and applications with a 
capacity and usability wl^h were even do^ to the level reached by larger data-pmoessing 
devices, like desktop or laptop computers. It is also impossible to include conventional 
storage means, Hke hard disk, floppy disk on CD-ROM drivers, in hand-held devices. A 
solution of these problems may be a two-part data-processing device wherein inahand-hald 
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part tbere is only a part of necessary circuits and programs in addition to a display. The most 
of the circuits and programs are in another portable part held by the user, a wireless lick 
connecting this part to The haod-held part. The vrireless link may operate with IR or radio 
frequencies. In this kind of device, it is easy to realize at the same time a system which 
5 detects movement or position of the hand-held part in relation to the portable part The 
operation method of the user interface according to the invention may then he realized by 
means of this system. 

To the following, a further embodiment of the invention hased on detecting: 
accelerations is considered with reference Co Fig. 16. The device includes a multi-axial 
10 accelerometer and necessary circuits and programs for measurements. The device being 
switched an, it is monitoring accelerations and in phase 11 is interpreting that an 
acceleration above a certain threshold is possihly an initial stage of a movement defined in 
the operation method of the riser interface- la response to this, timers TD1 and TD2 are 
started in phase 12. TD1 sets a very short experimentally determined time of the aider of 
15 milliseconds from the detected start of the movement, in which the actual direction and 

magnitude of the acceleration specify the movement which the otser gives to the device. TD2 
sets a longer time expiring of which stops changing data on the display if no other causes for 
- stopping appeared- At die time of expiration of time TD1, the prevailing acceleration vector 
is measured in phase 13. hi phase 14, it jse xamTnal if a detected vector i^ a defined yypyr, 
20 Le. if it is corresponding to any movement defined in the operation method of ffae user 
interface. If not^ the operation returns hack to the beginmng 10 and phase 1 1 to monitor 
accelerations of the device. On the other hand, if a vector is a defined vector, data on the 
display is changed in phase 15, e_g. is browsed, zoomed, etc., according to a corresponding 
algorithm. As is described above, e.g. determination of the browsing speed of data on the 
25 display on the basis of the magnitude of an acceleration vector may be related to tins. In 

phase 16, it is also monitored af the same time if time TD2 is expired. If so, ibe operation is 
forwarded to phase 19 in which changing of data on the display is stopped and the procedure 
is finished in phase 20. If time ID 2 is not expired, it is monitored if any opposite vector to 
the detected defined vector or any other stopping signal, like a push of the display, appear in 
30 phase 17. If an opposite vector or a stopping signal are detected in phase 1 changing data 
con the display is stopped in phase 1 9 aaid the procedure is finishocL If no opposite vector or 
other stopping signals are detected in phase 18, the operation returns hack to monitor above 
mentioned issues. 

The operation method of the user interface of the invention may be realized also in 
35 other ways than by irtilizing acceleration measurements. Any technique by which changes of 
place and position of the device may be measured oiay be applied A sohition which may be 
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contemplated Ls technique io winch a place and orientation of the device are detects by 

sending pulsed^ 

this technique. TxjOi aplace aodp 

fully detected. 

A possible solution Ls also the technique an whi ch so caBed opti cal moose is based and 
ia which aroovemeat and speed thereof are detected by observing and analyzing any surface 
in the vicinity, which La this ease could he a surface an the user, fro example By combining: 
tfrtsvrilh distance measurancii^ charges of 

distance and relative nxFvem^ of a device may 

be detected. 

The invention may* of course, be realized in various ways. It may be contemplated, far 
example, that ibr making detcrt^ 

measured many times within a sboit dine window, and a defined movcroert is detected if 
any one of the measured vectocs meets the requirements for detection. Naturally* also a nunc 
complicated analysis of a movement based m successive 

he cuntemplaed. A movement of a device may be detected also with an axangemeo^ for 
example, in which distances are nieasrired between transm^ 
device* on one hand* and on the user, on the other band. 

A user interface according to the inversion may be realized according to use and 
necessary features of a brad-held device as a suitable combination of the embodiments 
presaged above, the riser interface Including several operation modes m which the same 
movement of the device may correspond to different changes of data on the display. As is 
stated atow in relation to so™ 
device for selecting different operation modes_ 

The invention may be vari^ accompanying claims. 
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Claims 

1. An operation method of the oser interface of a hand-held device, like palm-top or pocket 
computer, mobile phone, communicator or like; in which operation method data, on the 

5 display (2*) of the device (2) is changod in response to defined movements of the device in 
die three-dhxiensionaf space of use thuaeof and said movemoits of the device and the 
corresponding changes of data on the display are defined so that a rooveaiiGnt and a 
corresponding change of data on the display have a canse-ooasecpiejice relationship which is 
analogous to canse-consequemce relationships of the ftree-dinieosuHia] space of perception 
10 and action of a user (I), characterxBed in dial the operation method incbrdes an operation 
mode in which in response to lilting (Tfi, T7) the device (2) data (SI, CI* — - Ci, Go) on 
the display (2") is browsed (M4) to a direction coirespoiKhiig to the direction of tilting. 

2 . An. operatLOQ method of the user interface of a hand-held device according to claim I a 
15 chantcterraed in that one or mare of the following features are further inc hided in the 

operation mode: 

in response to turning (TS) the device a stack of data (S12) from data arranged 
virtually In stacks (SI 1 - SIS) on a circle surrounding a user of the device (2) is selected 
whereby the stack (S12) is browsed by tilting (Fl) the device into the direction of the static 
20 (S12) and away from it, respectively; 

in response to moving (Z) the device (2) forward or backward pttpendKcutariy 
to the display (2) data (6a) on the display is zoomed larger or smaller, respediv^dy; 

in res|Mnise to moving 
the plane of the display £7) thereof data (q) from data(Dll> larger than the display is 
25 selected; and 

in response to having a cursor (C) placed on the display (2 a ) and moving (N, E, 
S, W) the device (2 J essentially in the direction of Lhe plane of the display (2*) thereof the 
cursor (C) Is moved against the movement for selecting daia(01 - 09) oa the display as if 
by moving data on the display in the real world below the cursor keeping its place in the real 
30 worfdL 

3. An operation inethod of the us^ of a hand-held device according to claim 1, 
characterised in that the operation method further includes an operation mode in which in 
response to swinging (P) the device (2) like tarring page also data (8a, 8b, Be) oh the display 

35 (T) is changed in the way rorresponding to turning page. 
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4. An operation method of the user interface of a band-held device accordiag to claim I, 
characterised in that !he operation method forther includes an operation mode in which in 
response to tilting (T2) the device (2) data (D2) is moved cm the display (2*) from the 
direction of tilting towards a cursor (G). 

5. An operation method of the user interface of a hand-held device according to claim 1, 
characterized in that the operation method further includes an operation mode in -which in 
response to tilting (T4) the device (2) selection of data (DS) on the display (2/) is moved 
(M3^ D9) into the dinxtiba of tilting, 

6. An operation method of the user interface of a handheld device according to claim I, 
characterized in that the open£km mettod faith 

response to lilting (T3)the device {2} away from a usee data (D3) on the display (2*) is 
enlarged (Z2) ami/ or in response to filling (T4) the device (2) towards a user data (DA) (T) 
is reduced (Z3y 

7. An operation method of the user interface of a hand-held device according to claim I, 
characterized in that the operation method further includes an operation mode in which 
displayed data is a desired part of a panorama picture (9a) and in response to taming the 
device (2) displayed data (9b) is chaaged so that the ciange of the viewing direction and the 
displayed part of the panorama picture (9a) correspond to turning. 

& An operation method of fberaer interface of a hand-heM ] y 
ekarai^erized in 3iat (be operation method furrier includes an operation mode in which 
displayed data is a desired part of a panorama picture car 3D image daia(Rf, R2) 3D and 
data (D 12) dexcri!ringa thing is placed io the image dalaand in rc^wiseto taming the 
device (2) hoth the paiMxama picture data or 2D image data (Rl, R2) and 3D image dala 
(D12\ D12") are changed on the display (2*) to correspond to the viewing direction (AI, 
A2) corresponding to the orientation of the device (2). 

9. An operation method of the user interface of a hand-held device according to claim l a 
characterized irv ftmt in response to stopping or significant slowing (1 8) of a movement 
changing of data op the display is stopped. 
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ia An operation rae&od of Ibe iiser interface of a hand-held device according to claim I, 
characterized m that changing of data on the display is stopped by giving a sigoaJ to the 
devioe hi oftff way thasity means of movemoit of die device. 
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,- (57)Abstiact 

An operation method of a user Interface of a handhe Id device, like a palm-size or pocket- 
° size PC, mobile teiepharu^ communicator oar equivalent, in which metho d data on a display 

of the device Is changed for finding or s 
? 5 - So) cm the display (2 1 ) of the device is changed responsive to certain movcanents (F) of the 

? d^iceinAthiw-dim^ 

: Ijje corresponding changes of the dala on the display are defined so that a movement and a 

COTiej^anding cfcai^ 

analogous to cause-consequence relationships of Ibe three-dimeaskra^ space of perception 
10 and actum of a user (1)- 
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Fig. 6 
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Fig. 8 




Fig. 9 



8/12 




Fig. 10 
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Fig- 1 1 
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Fig. 14 
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BEGIN 
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START OF A 
MOVEMENT »S 
DETECTED 
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TIMERS TD1 
ANDTD2ARE 

STARTED 
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TD1 HAVING EX- 
PIRED, ACCELE- 
RATION VECTOR 
IS MEASURED 
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DISPiAYED DATA 
IS CHANGED 
ACCORDING TO 
AN ALGORITHM 
CORRESPONDING 
TO THE VECTOR 
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MONITORING 
OPPOSITE 
VECTOR OR 
STOP SIGNAL 




YES 
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DATA ON THE 
DISPLAY IS 
STOPPED 
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END 



Fig. 16 



